Add --attachments flag to download issue and PR attachments

Fixes #399

Problem

User-uploaded attachments (images, videos, documents) referenced in issues and PRs are not currently backed up. Only their URLs are preserved in the JSON files. If a repository or account is deleted, these attachments are permanently lost - the URLs become inaccessible, and critical context like screenshots, diagrams, and documentation is gone forever.

Solution

The solution turns out harder than you would think! There is a lot to do in order to determine attachments and then get them. However after doing all that, we end up with something as simple as:

Adds --attachments flag to download binary attachments alongside issue/PR data.

Makes it look so easy!

What Qualifies as an Attachment?

GitHub's API doesn't have an "attachment" concept - it's a user behavior pattern. Through analysis of real-world repositories, attachments are defined as:

• Found outside code blocks (not wrapped in backticks)
• User-uploaded files hosted on GitHub's CDN domains
• Matching known URL patterns for GitHub attachments

Note this approach can captures bare URLs pasted by users, not just markdown image syntax. Occasional false positives, such as example URLs like https://github.com/user-attachments/assets/abc12345-6789-4def-ghij-klmnopqrstuv might fail with HTTP 404. Any failure is logged transparently in the manifest file.

Implementation

• Extracts attachment URLs from issue/PR body and all comment types
• Downloads attachments using HTTP requests with redirect handling
• Determines file extensions from Content-Disposition headers and redirect URLs
• Creates per-issue/PR manifest.json files with complete download metadata
• Integrates seamlessly into existing retrieve_issues() and retrieve_pulls() functions
• Respects --skip-existing: Loads existing manifest.json and only downloads new attachments (URLs not already in manifest)
• Saves to issues/attachments/{issue_number}/ and pulls/attachments/{pull_number}/ directories
• Handles conflicting filenames in same issue/PR

Technical Details

URL Format Support:

• Repo files: github.com/{owner}/{repo}/files/* (filtered to current repo only)
• Repo assets: github.com/{owner}/{repo}/assets/* (filtered to current repo only)
• Modern: github.com/user-attachments/{assets,files}/*
• Legacy: user-images.githubusercontent.com/* and private-user-images.githubusercontent.com/*

Smart URL Extraction:

• Strips markdown code blocks (fenced ``` and inline `) before searching for URLs
• Prevents extracting URLs from code examples and documentation snippets
• Extracts URLs from markdown text, HTML, and plain text
• Cleans trailing punctuation from URLs (periods, commas, etc. from sentences)

Cross-Platform Compatibility:

• Uses HTTP Content-Disposition headers for file extension detection
• No external dependencies - uses only Python stdlib (urllib, json)
• Works on Windows, macOS, and Linux

Private Repository Support:

• Supports both public and private repositories
• Modern github.com/user-attachments/* URLs require authentication for private repos
• Legacy public CDN URLs (user-images.githubusercontent.com) don't need/accept authentication
• Conditional authentication logic prevents HTTP 400 errors on public CDN URLs
• Uses existing token authentication (works with --token, --keychain, etc.)
• Tested successfully on private repositories

Incremental Backup Support:

• --skip-existing works intelligently with attachments
• Loads existing manifest.json and only downloads NEW attachment URLs
• Smart retry logic: skips successful downloads and permanent failures (404, 410, 451), retries transient failures (5xx, auth errors, network timeouts)
• Crucial for incremental backups: when new comments/attachments are added to old issues, only new attachments are downloaded
• Preserves existing download history (both successes and failures) in manifest
• Example: An issue has 3 attachments backed up. User adds a 4th attachment. Next backup with --skip-existing only downloads the new one, preserving the existing 3 and adding the 4th to the manifest.

Progress Logging:

• Logs attachment downloads with logger.info() messages matching existing tool patterns
• Example: 2025-11-02T22:06:42.152: Downloading 1 attachment(s) for issue #42
• Download failures logged as warnings with HTTP status codes
• Integrates seamlessly with existing verbose output

File Conflict Handling:

• Detects when multiple attachments have the same filename or conflict with the manifest
• Uses counter suffix pattern: report.pdf → report_1.pdf → report_2.pdf
• Original filename record in manifest data
• Protects manifest.json. Any conflicting attachment is renamed (manifest_1.json)
• During testing, found 22 issues/PRs in pypdf with filename collisions

Manifest Files:
Each issue/PR with attachments gets a manifest.json. For example the manifest for py-pdf/pypdf#1034 is:

{
  "issue_number": 1034,
  "issue_type": "issue",
  "repository": "py-pdf/pypdf",
  "manifest_updated_at": "2025-11-02T14:21:51.647658+00:00",
  "attachments": [
    {
      "url": "https://github.com/py-pdf/PyPDF2/files/9001047/test_google_sheet.pdf",
      "success": true,
      "http_status": 200,
      "content_type": "application/pdf",
      "original_filename": "test_google_sheet.pdf",
      "size_bytes": 15350,
      "downloaded_at": "2025-11-02T14:21:47.965316+00:00",
      "error": null,
      "saved_as": "test_google_sheet.pdf"
    },
    {
      "url": "https://user-images.githubusercontent.com/1658117/176374973-baf90c95-a3bd-4a8a-9823-5fbbe7f31f69.png",
      "success": true,
      "http_status": 200,
      "content_type": "image/png",
      "original_filename": "176374973-baf90c95-a3bd-4a8a-9823-5fbbe7f31f69.png",
      "size_bytes": 156036,
      "downloaded_at": "2025-11-02T14:21:49.943060+00:00",
      "error": null,
      "saved_as": "176374973-baf90c95-a3bd-4a8a-9823-5fbbe7f31f69.png"
    }
  ]
}

Testing

Tested on 2 major repositories with production-scale data:

[1] Availability rate: Percentage of attachment URLs that were successfully downloaded. Failures are typically deleted attachments, expired JWT tokens, or example URLs of github attachments outside of a code block. These don't show up on the web version of the issue either. See "About Attachments" in the README for further details.

[2] The 81.8% rate for python-github-backup is expected: the 2 failed downloads are fake example URLs (e.g., https://github.com/user-attachments/assets/abc12345-...) posted in issue #399 to demonstrate URL patterns. They were correctly extracted and attempted, returning 404 as expected for non-existent URLs.

Cases Tested:

• Multiple attachments per issue (up to 92 in a single issue tested)
• Mixed content types (PDF 57%, PNG 29%, JPEG, text, ZIP, GIF, SVG, CSV, DOCX)
• Issues with no attachments (graceful skip)
• Modern user-attachments URLs (with and without extensions)
• Legacy CDN URLs (user-images.githubusercontent.com)
• Repo file URLs github.com/{owner}/{repo}/files/* (filtered to current repo only)
• Repo assets URLs github.com/{owner}/{repo}/assets/* (filtered to current repo only)
• Cross-repository URL filtering (links to other repos correctly filtered out)
• URLs in code blocks (correctly filtered out)
• URLs with trailing punctuation (cleaned automatically)
• Deleted/404 attachments (graceful failure, logged in manifest)
• Expired JWT tokens (graceful failure, logged in manifest)
• Example/fake URLs (correctly attempted and logged as 404)
• Private repository support (authentication works correctly)
• Incremental backups with --skip-existing (only downloads new attachments)
• Smart retry logic (skips permanent failures 404/410/451, retries transient errors)
• Manifest generation and structure validation (1,044 manifests validated)
• Large-scale testing (2,214 attachments successfully downloaded)
• Filename collision handling (22 issues with collisions detected, all files preserved with unique names)

Cross-Platform:

• Tested on macOS
• Tested on Linux
• Tested on Windows

Backward Compatibility

• New optional flag, default off
• No changes to existing backup format
• No new dependencies
• Works with all existing flags and options

Files Changed

• github_backup/github_backup.py (+637/-3): Core attachment download implementation with manifest tracking, collision detection, and
  smart retry logic
• README.rst (+30/-0): Documentation for --attachments flag and manifest format

Thank you for considering this PR as an addition to an already great tool!